Skip to content

[Cleanup] Small fixes to the multi-modal models - #827

Open
bohnstingl wants to merge 3 commits into
torch-spyre:mainfrom
bohnstingl:multi_modal_edits
Open

[Cleanup] Small fixes to the multi-modal models#827
bohnstingl wants to merge 3 commits into
torch-spyre:mainfrom
bohnstingl:multi_modal_edits

Conversation

@bohnstingl

Copy link
Copy Markdown
Collaborator

Description

Refine the recent Pixtral/Ministral multimodal enablement without expanding the supported model surface:

  • Route all CPU↔Spyre transfers through convert(), including layout-aware transfers.
  • Prepare tiled convolution weights once after model loading instead of lazily in the forward path.
  • Consolidate repeated tensor-tree conversion logic in _SpyreModelWrapper.
  • Reuse the shared conversion helper for model input, multimodal input, embedding-merge, logits, and model-output boundaries.
  • Preserve the existing native Mistral-format Pixtral behavior and linear fast path.

Related Issues

Relates to #207

Test Plan

This PR does not introduce any functional change.

cc @nikheal2

Signed-off-by: Thomas Ortner <boh@zurich.ibm.com>
@bohnstingl
bohnstingl requested review from a team, dilipgb and yannicks1 as code owners September 9, 2026 22:04
@bohnstingl
bohnstingl requested a review from nikheal2 September 9, 2026 22:05
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! Thank you for contributing.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, run ./format.sh.
Now you are good to go 🚀.

We also recommend installing prek and configuring it to check your code before every local commit.

…lti_modal_edits

Signed-off-by: Thomas Ortner <boh@zurich.ibm.com>
Signed-off-by: Thomas Ortner <boh@zurich.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant